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Abstract 

In this Letter we describe a search for lepton flavor violation (LFV) in the bottomonium system. 
We search for leptonic decays T(nS') — > [it (n = 1,2 and 3) using the data collected with the 
CLEO III detector. We identify the r lepton using its leptonic decay u T v e e and utilize multidimen- 
sional likelihood fitting with PDF shapes measured from independent data samples. We report 
our estimates of 95% CL upper limits on LFV branching fractions of T mesons. We interpret our 
results in terms of the exclusion plot for the energy scale of a hypothetical new interaction versus 
its effective LFV coupling in the framework of effective field theory. 
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The subject of this Letter is a search for lepton flavor violating (LFV) bottomonium 
decays T(nS) — > /xr (n = 1,2 and 3). Such decays are predicted by various theoretical 
models that allow tree-level flavor-changing neutral currents (FCNC), including, e.g., im- 
parity violating and large tan 3 SUSY scenarios, leptoquarks, and other models inspired by 
the idea of grand unification p], [2} . Our search is motivated by the discovery of large mixing 
between the second and the third generations in the neutrino sector 0]. 

The conservation of lepton, lepton flavor, and baryon quantum numbers in the standard 
model (SM) is due to accidental global symmetries of its Lagrangian. All such symmetries 
should be violated at higher energies, where we expect the emergence of a gauge group of the 
higher-order symmetry that presumably describes fundamental interactions at the energy 
scale of grand unification. The search for beyond the standard model (BSM) physics in 
low-energy processes is facilitated by parameterizing such BSM physics, without explicitly 
invoking its unknown dynamics, in the framework of the Wilson operator product expansion 
(OPE) and effective field theory. The large lepton mass hierarchy and dimensional analysis 
suggest that the effects of BSM physics are most likely to be observed in transitions that 
involve heavy quarks, muons, and r leptons. In the OPE the effects of BSM physics in 
decays T(nS) — > fir are expressed by the four-fermion diagonal operators 0,0] that respect 
the full electroweak SM gauge group SU (2)l®U (l)y and contribute to the SM Lagrangian 
as 

C eS = C SM + ^(/T>)(6 7 ^), (1) 

where T M is a vector (7^) or an axial (7^75) current or their combination, A is the scale of 
BSM physics and is the effective LFV coupling of the new gauge symmetry associated 
with BSM. 

Previously, we searched for LFV in B meson decays jq], while the BES experiment 
searched for LFV in J/ijj decays [7|. Those two analyses probed the BSM contributions 
parameterized by the operators (pTr)(bTd) (T = 75,757^) and (pT M r)(c7 M c), respectively. 
In the analysis presented in this Letter we probe the four-fermion operators (pT m t)(&7 m &). 

The CLEO III detector, centered on the interaction region of the Cornell Electron Storage 
Ring (CESR), is a versatile multi-purpose particle detector Relevant components of the 
apparatus include a nearly Air tracking volume surrounded by a Ring Imaging Cherenkov 
Detector (RICH) 0], an electromagnetic CsI(Tl) crystal calorimeter, and a muon identifi- 
cation system |lOj consisting of proportional wire chambers that provide two-dimensional 
position information. The tracking volume, located inside an axial magnetic field of 1.5 T, is 
instrumented with a 47-layer wire drift chamber and a four-layer silicon strip detector that 
allow us to measure the positions, momenta, and specific ionization energy losses (dE/dx) of 
charged particles with momentum resolution of 0.35% (0.86%) at 1 GeV/c (5 GeV/c) and a 
dE/dx resolution of 6%. The calorimeter, first installed in the CLEO II detector [ll|, forms 
a cylindrical barrel around the tracking volume and has resolution of 2.2% (1.5%) for 1 GeV 
(5 GeV) photons and electrons. The calorimeter, just inside the magnet coil, is followed by 
Fe flux-return plates interleaved with three layers of the muon identification system. 

We search for non-SM leptonic decays T(nS) — > fir (n = 1,2 and 3) using the data 
collected with the CLEO III detector. We identify the r lepton using an electron from its 
leptonic decay v T v e e. We use data samples that contain 20.8, 9.3, and 5.9 million Y(IS'), 
T(25), and T(3S) resonant decays, respectively [H, T3|. Integrated e + e~ luminosities of 



these signal data samples are 1.1 fb l , 1.3 fb 1 , and 1.4 fb 1 . We use the T(4S) (6.4 fb x ) 
and hadronic "continuum" (2.3 fb -1 collected 60 MeV below the T(4S*) energy) data to 
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measure the shapes of probability density functions (PDFs) and resolution parameters used 
in maximum likelihood (ML) signal fits described later in this Letter. We also use the T(4S) 
and continuum data to verify the overall reconstruction and trigger efficiency and to estimate 
systematic errors. 

The signature of our signal is a muon with p^j E heiim ps 0.97 and an electron from the decay 
of the r lepton. We select events with two reconstructed tracks of opposite electric charge. 
One track is identified as a high-quality muon candidate by requiring that it penetrate five 
hadronic interaction lengths. The other track should satisfy electron identification criteria 
by requiring a ±3cr consistency with the theoretically-predicted dE/dx contribution and 
0.85 < E/p < 1.10, where E is the energy reconstructed in the region of the electromagnetic 
calorimeter matched to the projection of electron's track of momentum p. Electron and 
muon candidates should not also be identified as the candidates of the other lepton species. 
The beam-energy normalized momenta of the muon and electron candidates, x = p^/^beam 
and y = p e j E heam , are required to be within the ranges 0.87 < x < 1.02 and 0.10 < y < 0.85. 

The geometric acceptance of tracking is ~ 86% for two tracks. Track reconstruction 
efficiency for the signal is 83% in the acceptance region. The muon system coverage is 
84% of the solid angle and the efficiency of muon identification in that region is 92% per 
muon when its charged track is reconstructed. Electron identification is 95% efficient, due 
to the calorimeter's angular acceptance. The trigger for signal events in fiducial region of 
the detector is 93% efficient. The efficiency for selecting events in the x and y regions (after 
applying all other criteria) is 95%. Trigger and reconstruction efficiency for the signal is 
50%. Its product with the £>(r — > v T v e e) = (17.84 ± 0.05)% yields an overall efficiency 
of 8.9%. 

We do not expect to find LFV in the T(4S) and hadronic continuum data used to calibrate 
our analysis method. Even if LFV BSM physics, e.g., quantum gravity, becomes strong at a 
TeV energy scale, LFV would occur in dilepton decays of the T(4S*) at a much smaller rate 
than in decays of lower-mass bb resonances, because the products of their production cross 
sections and SM dilepton partial widths are significantly larger than that for the T(AS). 
The BaBar experiment has recently published an upper limit (UL) for a(T/j,)/a(ee) at the 
T(4S*) energy [l5j. Their UL suggests that less than 3 LFV events would be observed in our 
calibration data. We show the distribution of y versus x for our calibration data in Fig. [U^a) 
and the projection onto the axis x in Fig. [Tib). 

According to our studies, confirmed by Monte Carlo (MC) simulation for QED processes, 
three backgrounds arising from fi and r pairs contribute to the distributions shown in Fig. [IJ 
The fi pairs contribute in two ways, through radiative processes, and, also, when one muon 
decays to an electron in flight. The first contribution from pairs includes QED radiation at 
the vertex and hard bremsstrahlung in the detector. Such events satisfy our selection criteria 
when a radiative photon matches the muon track's projection to the calorimeter and muon 
identification fails. Such events cluster around y = 0.53 because of the E/p requirement, 
where E, for such background events, is the energy of radiative photon (^ E hcam /2) combined 
with a small amount of energy (^ 0.2 GeV) deposited by the muon in the calorimeter. 

The second, and less frequent background from pairs appears when one muon decays in 
flight. This results in the actual electron detected in the calorimeter. Such events cluster near 
x = 1 but scatter in y between 0.10 and 0.85. Both background contributions from fi pairs 
differ from the hypothetical signal in T(nS) (n = 1,2 and 3) data. The high-momentum 
background muon is most often produced at beam energy, x = 1 (though radiative processes 
introduce a long tail in the x shape for this background), while the signal muon peaks at x = 



4 



0.965,0.968, and 0.970 for the T(IS'), T(2S), and T(3S f ), respectively. Also, when a muon 
mimics an electron, the E/p and dE/dx distributions differ from those we expect for the real 
electrons. While the dE/dx measurements do not have sufficient resolution to discriminate 
between electrons and muons on an event by event basis in the relevant momentum region, 
namely, around 2.5 GeV, discrimination between the signal and backgrounds on a statistical 
basis is possible. 

The production of r pairs represents an irreducible background to our signal when both r 
leptons decay leptonically, one to an electron and the other to a muon. The only variable that 
discriminates our signal from this background is x, the beam-energy normalized momentum 
of the signal muon candidate. 




0.87 0.92 0.97 1.02 0.87 0.92 0.97 1.02 

X = P^i / ^beam 

FIG. 1: (a) The scatter plot of y versus x and (b) its binned x projection for calibration data. 
The location of the hypothetical signal peak is indicated by the arrow, where the width of the 
horizontal bar at its tip is ±cr(x). 



To estimate the number of LFV decays in T(nS) (n — 1,2 and 3) data we subject the 
events that pass the selection criteria to four-dimensional unbinned extended ML fits. For 
each probed data sample we maximize the likelihood function 

i / 4 \ JV 4 

L = — exp nEW({*}« {«U ( 2 ) 

\ 3 J i 3 

where N is the total number of data events in the fit; % is the index for these events; j is the 
index for fit contributions (the signal and the three backgrounds); {z}i is the vector of the 
four variables x, y, dE/dx and E/p for event i; Nj is the fit parameter that corresponds to 
the numbers of events for fit contribution j; and Vj is the four-dimensional PDF with shape 
parameter vector {a}j for fit contribution j. 

We utilize calibration data to find approximations for the PDFs, the values of their shape 
parameters and respective matrices of systematic errors. The correlations among the vari- 
ables, especially important for the /z-pair backgrounds, are included in the respective PDFs. 
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To take into account initial state radiation, we parameterize the x shape of the /z-pair back- 
ground using a Gaussian with a long asymmetric tail. The E/p shape for real electrons is 
also parameterized by such a Gaussian. The x shape for the r-pair background is parame- 
terized by a first order polynomial smeared by Gaussian detector resolution measured using 
the data. The E/p shape for the muon matched with a radiative photon in the calorimeter, 
therefore misidentified as the signal electron candidate, is approximated by a first order 
polynomial. The beam-energy normalized electron momentum, y, is parameterized by a 
second order polynomial for the signal, r pairs, and \i pairs when one muon decays in flight. 
For radiative \i pairs the shape of y is approximated by Gaussian with a long asymmetric 
tail whose mean depends on E/p. We approximate dE/dx shapes by Gaussians. The signal 
x shape is approximated by a Gaussian with the resolution a(x) = 0.86% ± 0.03%, which 
we measured using radiative fi pairs. We studied the performance of our fitting method 
by mixing signal toy MC events with calibration data. No biases were observed in these 
studies. We also verified our results by rejecting events where the signal electron and muon 
candidates are back to back. Such selection efficiently suppresses the /z-pair backgrounds 
but lowers the sensitivity to the searched-for LFV signal. To further verify the analysis 
presented in this Letter we performed a one-dimensional ML fit of the x distribution for 
events remaining after this selection, obtaining results consistent with the main analysis but 
with lower efficiency and reduced significance. 

Systematic uncertainties in our analysis arise from several sources. The largest contribu- 
tions to the error on the efficiency come from the trigger (5%), event selection (4%), track 
reconstruction (3% for two tracks), muon identification (2%), online event preselection (2%), 
signal MC statistics (2%), software trigger (1%), and electron identification (1%) uncertain- 
ties. The overall systematic error on the efficiency is 8%. To verify this error estimate we 
measured the partial cross section for r-pair production in the region 0.65 < x < 0.95 using 
calibration data where no signal and no contamination from \x pairs are expected. Properly 
scaled up to the total cross section for r-pair production at 5 GeV, our measurement agrees 
with the expected 0.92 nb within 4%, while the statistical uncertainty of this measurement 
is 5%. 

The uncertainty in the y shape and in the efficiency of y region selection for the signal 
are determined by the uncertainty in r polarization. The polarization of r is well-defined for 
QED processes but is model-dependent for BSM contributions. The efficiency of the V + A 
(V — A) hypothesis, when the electron from r decay is boosted forward (backward), is 3% 
lower (higher) than in the case of an unpolarized r. We use the unpolarized r efficiency in 
the analysis and estimate the systematic error in the efficiency of the y region selection that 
arises from r polarization uncertainty to be 3%. Significantly larger systematic errors (up 
to 15%) are associated with the uncertainties in PDF shape parameters in ML fitting. To 
convert signal yields to LFV branching fractions we also take into account the 2% uncertainty 
in T statistics. 

To determine parametric dependence of the likelihood function on the signal yield (and 
LFV branching fraction), we integrate the likelihood function over the other three fit pa- 
rameters, i.e., the numbers of background events. We take the uncertainties in PDF shape 
parameters into account by performing 1000 ML fits for each data sample using the PDF 
shape parameters determined from T(4S) and continuum data but varied according to Gaus- 
sian uncertainties in their values in each fit. In addition, to obtain the likelihood distribution 
for the LFV branching fraction we vary the efficiency and the number of T mesons in each 
of these fits according to their Gaussian uncertainties. The resulting distribution of the 
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likelihood function is the sum of such individual distributions of likelihoods, each obtained 
with its own set of PDF shape parameters, the efficiency and the number of T decays. This 
technique takes into account the systematic error on the LFV branching fraction arising 
from the uncertainties in the PDF shape parameters and results in widening the likelihood 
distribution. 
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FIG. 2: (a) The binned x projection of the results of the ML fit to T(IS') data (points with the 
error bars). Solid line indicates the result of the fit, shaded areas show r-pair, /x-pair and signal 
contributions to the fit. Dashed line shows the hypothetical signal of 100 LFV events superimposed 
on the result of the fit. (b) The distribution of the likelihood function versus branching fraction 
for LFV decay T(15) -» fir. 



Our largest signal sample with relatively smaller QED background is T(1S) data. We 
show the binned x projection of the results of our four- dimensional unbinned ML fit to this 
sample in Fig. [2fa). The final distribution of the likelihood versus LFV branching fraction 
for leptonic decay T(IS') — > fir is shown in Fig. EJ^b). To estimate the 95% CL Bayesian UL 
on this branching fraction we integrate the likelihood function for positive (i.e. physical) 
values of the branching fraction and find the value that correspond to 95% of the area. We 
apply the same technique to the T(2S) and T(3S) data and show our results for the 95% 
CL ULs on the branching fractions for LFV decays of T mesons in Table [B 

Effective field theory allows one to relate the dilepton and LFV branching fractions of T 
mesons to the scale A of LFV BSM physics 0, 111 using 

r(T(nS) -> fir) 1 fa N y ( M(T(nS)) \ A 

r(T(n5) -»• mi) 2e 2 b \a J { A J ' 1 } 

where e\> is the charge of the b quark, M(T(nS)) is the mass of vector meson T(nS) and a is 
the fine structure constant. We show 95% CL lower limits (LL) on the BSM energy scale A 
assuming = 1 in Table [H This table also shows other quantities necessary for estimating 
A. 
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T(15) T(25) T(35) 



Mass (GeV/c 2 ) 9.46 10.02 10.36 

AT decays (millions) 20.8 9.3 5.9 

T(T -> W ) (keV) 1.252 0.581 0.413 

r(T) (keV) 53.0 43.0 26.3 

B(pn) (xl0~ 3 ) 23.6 13.5 15.7 

B(jj,t) (95% CL UL, xlO" 6 ) 6.0 14.4 20.3 

B(fiT)/B(nn) (95% CL UL, xl0~ 3 ) 0.25 1.1 1.3 

A (95% CL LL, TeV, a N = 1.0) 1.30 0.98 0.98 

TABLE I: Information necessary to interpret our results in terms of BSM physics scale A and 
coupling qzn . We assume lepton universality and use our results for dielectron partial widths of T 
mesons [161 ] . Full widths are according to the PDG summary [P 



To estimate the lower limit on the scale of BSM physics and to produce the exclusion 
plot of A versus ajy we combine our signal datasets by taking the product of individual 
likelihood functions obtained for each dataset before taking into account the systematic 
errors associated with the uncertainties in the overall reconstruction and trigger efficiency, 
PDF shape parameters and T statistics. In the product of the likelihood distributions each 
distribution is represented by 

a| = B(?(nS) -> fir) 2eW m 
A 4 B(T{nS) -.////) (M(T(nS))) 4 {) 

We show the resulting combined likelihood function in Fig. [3^a). We use this figure to 
estimate the 95% CL LL on the scale of BSM physics and to prepare the exclusion plot 
shown in Fig. [3](b). In Fig. [3]^a) we show the 95% CL LLs obtained separately with Y(IS') 
and, also with all three signal data samples combined. 

The improvement from combining all signal data samples is small (A > 1.34 TeV using 
all data as compared to A > 1.30 TeV using the T(IS') data), because all three samples 
correspond approximately to the same amount of the integrated e + e~ luminosity and contain 
similar numbers of background QED events. The larger cross section for the production of 
T(15) makes this sample dominate our results for A. The slightly more (less) restrictive 
limits on LFV branching fractions (by 3%) and A (by 1%) could be obtained assuming pure 
V — A (V + A) BSM interaction for which the efficiency is 9.2% (8.6%). Our interpretation 
of the LFV results from the BES experiment 0], A > 0.49 TeV at 95% CL, should not be 
compared with our results directly, because these two analyses probe different operators. 
Finally, the lower limits on A estimated [E| from the decays of B mesons are much more 
constraining, of the order of hundreds of TeVs, than the estimate obtained in our analysis. 
However, such analyses probe non-diagonal operators, where the source of possible BSM 
contribution is not necessarily the same as in the analysis presented in this Letter. 

To conclude, we searched for leptonic decays T(nS) — > \it (n = 1,2 and 3) predicted 
by various LFV BSM scenarios that would break the accidental lepton flavor symmetry 
of the SM. We estimate 95% CL ULs on B(T(nS) -> fir) to be 6.0, 14.4 and 20.3 for 
n = 1,2 and 3, respectively, units of xl0~ 6 . In the framework of effective field theory we 
probed the contribution from the operators (jlT^r) (bj^b) and interpret our results in terms 
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a 2 N /A*[1 /TeV 4 ] a N 

FIG. 3: (a) The distributions of the likelihood functions versus o^/A 4 (95% CL ULs are shown 
assuming ajv = 1) and (b) the exclusion plot for A versus a at. 

of the exclusion plot for the energy scale A of some new BSM interaction and the strength 
of its effective LFV coupling and, assuming — 1, estimate the 95% CL LL on A to be 
1.34 TeV. 
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